WIP: Test files with Elasticsearch json logging #9988
Conversation
es_deprecation.log - started from binary - contains a json log line with type deprecation es_docker_happy_startup.log - started from docker - successful elasticsearch startup es_docker_startup_and_deprecation.log - started from docker - log lines containing successful startup (type main) mixed with log line of type deprecation es_docker_startup_bootstrap_fail.log - started from docker - log lines from startup failing due to bootstrap fail (not a json in docker) es_happy_startup.log - started from binary - successful elasticsearch startup es_startup_failure_boostrap.log - started from binary - elasticsearch failing to start due to bootstrap failure es_startup_shutdown_due_to_exception.log - logs from DieWithDignity where elasticsearch fail to start due to exception es_json_logs_tests.log - logs from a junit test - mix of logs with: - stacktrace with a cause - stacktrace with exception message as json - message with json - messages with multiple logger level
pgomulka
added
in progress
|
This is work in progress to integrate beats with new elasticsearch json logging |
|
Pinging @elastic/stack-monitoring |
|
Thanks for these samples, @pgomulka! |
| @@ -0,0 +1,55 @@ | |||
| {"type": "main", "timestamp": "2019-01-10T10:38:14,990+0100", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "distribution_run", "node.name": "node-0", "message": "using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [187gb], net total_space [465.6gb], types [apfs]" } | |||
| {"type": "main", "timestamp": "2019-01-10T10:38:14,993+0100", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "distribution_run", "node.name": "node-0", "message": "heap size [989.8mb], compressed ordinary object pointers [true]" } | |||
| {"type": "main", "timestamp": "2019-01-10T10:38:14,996+0100", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "distribution_run", "node.name": "node-0", "message": "node name [node-0], node ID [BboLkxHbQAW5IyqyqAfBLA]" } | |||
There was a problem hiding this comment.
It looks like the node.id is known here (BboLkxHbQAW5IyqyqAfBLA). Any reason it couldn't be extracted into a JSON property?
There was a problem hiding this comment.
good spot, but I don't think that with current ES infrastructure code I can access that value earlier. I need to wait for a internal clusterStateUpdate to pass that value to log4j code
Will confirm that
|
In all the JSON logs there is a |
|
@ycombinator |
|
I have updated the logs according to latest config in a PR. |
|
@pgomulka If you get a chance, could you add some samples for search and indexing slowlogs as well? |
|
@ycombinator sure - just uploaded (with stats field too) |
|
closing as this was just a draft |
es_docker_happy_startup.log - started from docker - successful
elasticsearch startup
es_docker_startup_and_deprecation.log - started from docker - log lines
containing successful startup (type main) mixed with log line of type
deprecation
es_docker_startup_bootstrap_fail.log - started from docker - log lines
from startup failing due to bootstrap fail (not a json in docker)
es_happy_startup.log - started from binary - successful elasticsearch
startup
es_deprecation.log - started from binary - contains a json log line with
type deprecation
es_startup_failure_boostrap.log - started from binary - elasticsearch
failing to start due to bootstrap failure
es_startup_shutdown_due_to_exception.log - logs from DieWithDignity
where elasticsearch fail to start due to exception
es_json_logs_tests.log - logs from a junit test - mix of logs with: